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Foreword 


This  effort  was  funded  by  the  Operations  Analysis  Division  (OAD)  under  program 
HQMC  AVN  APX.  The  overarching  objective  of  this  research  effort  is  to  develop  a  non- 
cognitive  profile  of  successful  US  Marine  Corps  Air  Traffic  Controllers  (ATC)  to  assist  in 
the  selection  and  screening  process  of  Marines  for  ATC  duty.  This  aim  was  realized  by  a 
three-phase  process,  as  discussed  in  the  previous  technical  report  titled  Suitability 
Screening  Test  for  Marine  Corps  Air  Traffic  Controllers  Technical  Report  (NPRST-TR- 
13-1 ).  The  current  report  concerns  the  implementation  of  Phase  II:  the  development  and 
validation  of  a  suitability  profile  of  success  in  the  Marine  ATC  Schoolhouse.  Derived 
from  scores  on  the  Navy  Computerized  Personality  Scales  (NCAPS),  the  suitability 
profile  was  used  to  predict  success  using  Schoolhouse  performance  data.  Utilizing  this 
type  of  non-cognitive  profile  can  greatly  increase  the  efficiency  of  the  Marine  Corps  ATC 
personnel  pipeline  and  relieve  the  burden  of  unproductive  training  through  detection  of 
Marines  who  are  and  are  not  likely  to  graduate  or  perform  satisfactorily  as  ATCs. 

The  authors  wish  to  thank  the  funding  sponsor,  and  project  officers  for  their 
assistance  in  this  project.  A  number  of  Marine  Corps  Air  Traffic  Control  instructors  at 
the  ATC-School  provided  invaluable  assistance  in  the  collection  of  questionnaire  and 
academic  data,  and  their  accessibility  greatly  enhanced  the  development  of  the 
validation  program  undertaken  in  the  present  study. 


D.  M.  CASHBAUGH 
Assistant  Chief 

Navy  Personnel  Research,  Studies  and  Technology 
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Summary 


Problem 

Entry  into  military  occupational  specialty  (MOS)  training  schools  requires  a 
minimum  score  requirement  on  the  Armed  Services  Vocational  Aptitude  Battery 
(ASVAB),  a  test  battery  that  assesses  cognitive  abilities  through  performance  in  reading, 
mathematics,  general  science,  as  well  as  basic  knowledge  about  electronics,  mechanical 
systems,  and  automotive-shop.  The  ASVAB  was  designed  to  increase  the  effectiveness 
and  efficiency  of  the  selection  procedure,  and  determines  that  incumbents  have  the 
necessary  cognitive  skills  to  handle  the  technical  aspects  of  training  and  job 
performance.  However,  the  current  Marine  Corps  Air  Traffic  Control  selection  process 
using  the  ASVAB  has  resulted  in  cognitively  proficient  applicants,  who  are,  nevertheless, 
classified  as  “unsuited”  for  ATC  duty  during  post-training  performance.  The  inclusion  of 
individuals  into  the  ATC  MOS  who  will  ultimately  be  unfit  for  this  position  is  an 
inefficient  use  of  time  and  resources.  Building  on  the  first  phase  of  the  research  effort, 
which  developed  and  validated  a  Marine  Air  Traffic  Control  Suitability  Screening  Test 
(MATC-ST)  on  Operational  Force  performance  metrics  (Walker,  Farmer,  &  Roberts, 
2012),  Phase  II  discusses  findings  from  validity  analysis  utilizing  Marine  ATC 
Schoolhouse  data.  Making  such  a  suitability  profile  will  greatly  increase  the  efficiency  of 
the  Marine  Corps  ATC  personnel  pipeline  and  reduce  the  burden  of  unproductive 
training  by  providing  early  detection  of  Marines  who  are  and  are  not  likely  to  perform 
satisfactorily  as  ATCs. 

Objective  or  Purpose 

Individuals  possess  a  variety  of  abilities,  preferences,  interests,  and  personal 
characteristics  that  should  be  useful  in  predicting  who  will  be  best  suited  for  different 
types  of  military  jobs  or  duties.  The  overarching  objective  of  this  project  is  to  capitalize 
on  these  individual  differences  by  developing  a  non-cognitive  profile  of  successful  air 
traffic  controllers  to  be  used  in  the  selection  procedures  for  Marine  Corps  ATC  duty.  To 
reach  this  objective,  test  scores  on  measures  of  non-cognitive  traits  have  been  used  to 
develop  and  validate  a  profile  (MATC-ST)  for  successful  ATC  performance  in  the 
operating  forces.  The  current  phase  of  this  project  seeks  to  validate  a  non-cognitive 
profile  in  ATC  performance  in  the  schoolhouse.  Validation  based  on  schoolhouse 
performance  data  seeks  to  minimize  ATC  academically-related  training  failures  and  to 
further  emphasize  the  utility  of  considering  non-cognitive  traits  in  conjunction  with 
ASVAB  standards  when  selecting  for  ATC  duty. 

Approach  or  Method 

Using  a  multifaceted  adaptive  personality  measure,  the  Navy  Computer  Adaptive 
Personality  Scales  (NCAPS),  the  authors  assessed  ATCs  in  the  supporting  establishment 
operating  forces  (7257,  7252,  7253,  7254  MOS)  and  Marine  ATC  students  (7251  MOS) 
on  19  different  individual  traits.  Through  both  concurrent  and  predictive  validity 
designs,  scores  on  the  NCAPS  were  correlated  with  measures  of  schoolhouse  academic 
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performance,  and  statistically  significant  traits  were  identified.  The  resulting 
hierarchical  regression  analyses  produced  a  subset  of  relevant  NCAPS  traits,  and  two 
profile  algorithms  were  created  to  significantly  predict  schoolhouse  performance.  These 
profiles  were  referred  to  as  the  Marine  Air  Traffic  Control  Suitability  Screening  Test  A 
(MATC-ST  A)  and  the  Marine  Air  Traffic  Control  Suitability  Screening  Test  B  (MATC-ST 
B). 

Findings  or  Results 

The  subset  of  NCAPS  traits  that  demonstrated  statistically  significant  prediction  for 
ATC  schoolhouse  performance  and  were  included  the  Marine  Air  Traffic  Controller  - 
Suitability  Screening  Test  algorithms  were:  Achievement  Motivation,  Initiative, 
Perceptiveness,  Social  Orientation,  and  Self-Relianee.  Hierarchical  regression  analyses 
suggest  that  the  non-cognitive  suitability  measure  adds  incremental  validity  above  and 
beyond  the  ASVAB  score,  and  accounts  for  29%  of  the  variance  of  ATC  schoolhouse 
performance  (R2=.2o8,  p  =  .541,  t  =  4.646,  p  <  .000).  Given  a  MATC-ST  score,  one  can 
predict  the  expected  level  of  success  an  individual  Marine  is  likely  to  exhibit  in  the  ATC 
schoolhouse  and  can  make  subsequently  valid  ATC  duty  selection  decisions. 

Conclusions 

The  Marine  ATC  Suitability  Screening  Test  is  a  valid  algorithm  derived  from 
dimensions  of  NCAPS  and  has  been  shown  to  be  predictive  of  success  for  Marine  ATCs 
in  the  Schoolhouse.  Those  with  higher  suitability  scores  performed  better  on  knowledge 
and  simulation  exams,  and  graduated  at  a  higher  rate  than  those  with  low  suitability 
scores.  Further  data  collection  and  analysis  following  the  ATC  schoolhouse  graduation 
to  7257  qualification  will  provide  confirmation  of  the  MATC-ST’s  predictive  validity, 
both  for  training  performance  and  ATC  job  performance.  Further  analysis  to  derive 
conservative  cut-scores  from  the  combined  results  of  the  concurrent  and  predictive 
validation  from  the  ATC  Schoolhouse  is  recommended.  If  the  MATC-ST  is  to  be  utilized 
for  selection/classification  it  is  recommended  that  it  be  used  only  in  addition  to  the 
ASVAB,  and  implemented  at  the  Military  Entrance  Processing  Station  (MBPS)  prior  to  a 
Marine  ATC  MOS  designation.  This  will  reduce  attrition/revocation  and  increase  the 
quality  of  Marines  selected  for  the  ATC  MOS.  MATC-ST  implementation  can  improve 
the  Marine  ATC  screening  process  through  better  schoolhouse  performance,  better 
operating  forces  performance,  and  increased  diversity  through  fair,  valid  screening 
improvements. 
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Introduction 


The  previous  technical  report  titled  Suitability  Screening  Test  for  Marine  Corps  Air 
Traffic  Controllers  Technical  Report  (Walker,  Farmer,  &  Roberts,  2012),  gave  a 
complete  introduction  of  the  research  involved  for  this  study.  As  a  review,  the  current 
Air  Traffic  Control  selection  process  utilized  by  the  US  Marine  Corps  is  being  reviewed 
as  a  result  of  an  unacceptable  number  of  Marine  Air  Traffic  Controllers  (ATCs)  being 
classified  as  “unsuited”  for  ATC  duty  as  defined  by  performance  and  behavioral 
expectations.  The  first,  most  important  step  in  creating  a  more  efficient  force  is  to 
improve  the  Air  Traffic  Controller  (ATC)  screening/selection  process.  This  research 
effort  focused  on  developing  a  non-cognitive  Air  Traffic  Controller  profile  that  could  be 
incorporated  into  a  model  for  their  selection.  Such  a  profile  would  allow  the  Marine 
Corps  to  identify  those  Marines  who  are  likely  to  be  successful  as  Air  Traffic  Controllers, 
those  who  need  a  greater  depth  of  screening,  and  those  who  are  likely  to  fail. 

During  Phase  I  of  this  research  study  the  focus  was  on  data  collection  and  analysis 
from  Marine  ATCs  in  the  operating  forces  and  within  the  ATC  schoolhouse.  Phase  I 
results  reported  significant  findings  within  the  Operating  Forces,  and  recommended 
further  predictive  validity  analysis  from  the  ATC  schoolhouse  data.  This  report  will 
discuss  Phase  II  of  the  study  and  includes  the  predictive  validity  analysis  from  the  ATC 
schoolhouse  data. 


Background 

Phase  I  of  this  study  was  a  concurrent  validity  study  utilizing  data  collection  and 
analysis  from  Marine  ATCs  in  the  operating  forces.  The  subset  of  NCAPS  traits  that 
demonstrated  statistically  significant  prediction  for  ATC  (operating  forces)  performance 
and  were  included  the  Marine  Air  Traffic  Controller  -  Suitability  Screening  Test  (MATC- 
ST)  algorithms  were:  Adaptability /Flexibility,  Vigilanee,  Empathy,  and  Self-Relianee. 
Hierarchical  regression  analyses  suggest  that  the  non-cognitive  suitability  measure  adds 
incremental  validity  above  and  beyond  the  ASVAB  score,  and  accounts  for  14%  of  the 
variance  of  ATC  job  performance  (R2=.i4i,  p=.375,  t=5.804,  p<.ooo).  Given  a  MATC- 
ST  score,  one  can  predict  the  expected  level  of  success  an  individual  Marine  is  likely  to 
exhibit  on  the  job  and  can  make  subsequently  valid  selection  decisions. 

At  the  completion  of  Phase  I,  data  collection  and  analysis  from  the  Marine  ATC 
schoolhouse  was  still  in  progress.  Phase  II  of  this  study  will  discuss  findings  from 
validity  analysis  utilizing  Marine  ATC  schoolhouse  data.  This  effort  considers  the  Armed 
Services  Vocational  Aptitude  Battery  (ASVAB)  standard  that  is  established  to  minimize 
ATC  academically  related  training  failures  in  conjunction  with  the  suitability  screening 
test  to  replicate  Phase  I  findings. 
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Method 


In  personnel  selection  and  classification  it  is  customary  to  develop  measures  that 
predict  job  performance  and/or  job  tenure.  Measures  given  to  job  applicants  need  to 
assess  the  knowledge,  skills,  and  abilities  necessary  for  successful  performance  in  a 
particular  job,  ideally  without  producing  adverse  impact  (large  mean  differences)  for 
racial,  ethnic,  or  gender  groups.  This  study  relied  on  a  mix  of  predictive  and  concurrent 
validity  designs  using  NCAPS  and  Marine  ATC  performance  measures.  A  concurrent 
validity  design  is  when  the  predictor  is  administered  to  participants  close  in  time  to 
when  the  criterion  (performance  measure)  is  collected  (e.g.,  on  the  job).  NCAPS  was 
administered  in  such  a  concurrent  validity  design.  A  predictive  validity  design  is  when 
the  predictor  is  administered  long  in  advance  to  all  applicants  who  have  yet  to  be 
selected  for  the  job,  and  thus  represents  a  more  realistic  setting  in  which  the  instrument 
will  ultimately  be  used.  The  ASVAB,  to  be  compared  with  NCAPS,  was  administered  in  a 
predictive  validity  design. 

The  validation  approach  chosen  for  this  project  was  a  criterion-related  strategy.  This 
is  traditionally  accomplished  by  obtaining  the  test  scores  of  job  applicants  as  predictors 
and  then  collecting  measures  of  these  same  individuals’ job  performance,  the  criterion 
(or  criteria  if  more  than  one  type  of  measure  is  collected).  The  predictor  test  scores  are 
then  related  statistically  to  how  well  individuals  perform  on  the  job  and,  if  successful, 
can  be  used  to  identify  the  most  qualified  candidates  for  the  position  by  predicting  how 
individuals  with  particular  test  scores  will  likely  perform.  This  validation  methodology  is 
one  of  three  validation  strategies  presented  in  the  Uniform  Guidelines  on  Employee 
Seleetion  Proeedures  (1978,  EEOC),  the  Standards  for  Edueational  and  Psyehologieal 
Testing,  and  the  Society  for  Industrial/  Organizational  Psychology’s  (SIOP)  Prineiples 
for  the  Validation  and  Use  of  Personnel  Seleetion  Proeedures  (1987). 

In  order  to  statistically  perform  this  validation  for  Phase  II,  measurements  of  job 
performance  and  NCAPS  scores  were  provided  for  Marines  currently  training  at  the 
Marine  ATC  Schoolhouse  (both  concurrent  and  predictive  data).  This  included  Marine 
ATCs  already  attending  the  ATC  schoolhouse  and  any  new  Marine  ATC  arrivals  at  the 
schoolhouse. 

Data  Collection 

Data  were  collected  from  Marine  ATC  students  (7251 MOS)  at  the  Air  Traffic 
Controller  Schoolhouse  in  Pensacola,  Elorida.  Data  collection  consisted  of  interviews, 
observations,  performance  measurements,  and  NCAPS  scores.  The  following  data 
collection  section  is  provided  as  a  review  from  Phase  I:  Suitability  Screening  Test  for 
Marine  Corps  Air  Traffic  Controllers  Technical  Report  (Walker,  Parmer,  &  Roberts, 
2012). 
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ATC  Schoolhouse 


The  14-week  school  consists  of  three  blocks  of  training  that  have  knowledge  based 
written  tests  and  performance  based  assessments.  Block  One  training  is  six  weeks  long, 
entirely  knowledge  based,  and  concludes  with  the  FAA  qualification  written  exam.  Block 
Two  consists  of  local  control,  ground  control,  and  flight  data  performance  based 
assessments.  Block  Three  consists  of  basic  radar,  ASR,  PAR,  and  Arrival  written  tests 
and  performance  based  assessments. 

The  schoolhouse  requires  a  70%  or  higher  grade  to  pass,  however  in  the  operating 
forces  an  80%  or  higher  is  required  to  receive  qualifications.  ATC  officials  view  ATC 
school  as  preliminary  and  expect  that  more  extensive  learning  occurs  with  On-the-Job- 
Training  (OJT),  the  platform  MOS,  and  other  required  operating  force  qualifications  to 
maintain  qualified  controller  status. 

Schoolhouse  Instructor /Staff  Interviews 

NPRST  researchers  met  with  school  officials  to  discuss  the  study  objectives  and 
methods,  and  the  aptitude  strengths  and  weaknesses  of  incoming  trainees.  Researchers 
were  given  tours  of  the  classrooms;  the  tower  simulator  and  other  computer  based 
classroom  training,  and  allowed  individual  time  to  interview  key  instructors  in  this 
process.  Arrangements  were  also  made  for  collection  of  performance  measures  for 
students. 

•  Marine  Corps  has  an  ASVAB  GT  (VE,  Verbal  +  AR,  Arithmetic  Reasoning  +  MC, 
Mechanical  Comprehension)  score  of  no  as  a  minimum  qualifier.  A  Marine 
recruit  is  then  designated  into  Aviation  and  disseminated  between  ATC,  Weather, 
and  UAV.  Instructors  also  confirmed  that  this  process  leads  to  motivational 
concerns  as  to  whether  Marines  selected  for  ATC  School  have  the  motivation  to 
be  there. 

•  According  to  instructor/Staff  interviews,  trends  were  reported  in  higher  Block 
One  student  failures  of  the:  General  Tower,  ATC  Terminal  Procedures,  and 
Emergency/Non-Radar  Exams;  higher  Block  Two  student  failures  of  the:  Local 
Control  Exam;  and  no  trends  reported  for  Block  Three  student  failures. 

•  Eor  performance  based  evolutions  (Blocks  Two  and  Three)  students  are  evaluated 
with  critique  sheets  that  are  filled  out  for  each  evolution  that  include  instructor 
comments.  The  Critique/Evaluation  sheets  are  based  on  the  position  being 
evaluated  (Local  Control,  Arrival,  etc.)  and  students  are  given  three  scores:  (S) 
satisfactory,  (N)  needs  improvement,  (U)  unsatisfactory.  Instructors  give 
recommendations  for  improvement  on  these  critique  sheets.  A  student’s 
performance  folder  is  collected  upon  graduation/drop  and  destroyed  after  two 
years. 

Other  comments  concerning  motivation  for  Marine  7251s  from  instructors  and  staff 
were: 

•  The  instructors  commented  on  attributes  besides  intelligence  that  make 
successful  7251s.  Some  mentioned  skills  like  multitasking,  such  as  “being  a 
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bartender”,  is  comparable  to  being  an  ATC.  Others  commented  that  skills  like 
achievement  motivation  and  perseverance  count  the  most  in  getting 
through  the  ATC  School  successfully. 

•  “Students  don’t  fear  authority.” 

•  “They  don’t  see  a  problem  with  quitting.  If  I  put  a  bell  on  the  quarterdeck,  they 
would  ring  it  here!” 

•  “They  don’t  see  a  problem  with  failing.  They  treat  it  like  a  video  game.  They  make 
a  mistake  and  they  want  to  push  reset  and  start  over.” 

•  Remediation  and  re-test  is  guaranteed  to  students  who  have  failed  an  exam. 
Instructors  noted  that  those  students  that  reach  out  for  help,  use  labs  outside  of 
class  time  and  that  ask  questions,  or  take  initiatives  to  approach  instructors  after 
class  are  the  most  successful  at  successfully  completing  ATC  School. 

Schoolhouse  Performance  Measures 

In  order  to  measure  performance  at  the  ATC  schoolhouse  we  collected  Grade  Point 
Averages  (GPA)  for  each  Marine  student  at  each  block  of  training.  In  the  context  of  this 
report,  GPA  is  a  composite  score  consisting  of  the  mean  average  for  each  block  of 
training.  GPAs  were  collected  at  the  end  of  Block  One,  Block  Two,  and  Block  Three  for 
each  Marine  ATC  student  (7251). 

NCAPS  at  Schoolhouse 


The  Navy  Computer  Adaptive  Personality  Scales  (NCAPS)  is  a  web-based  non- 
cognitive  measurement  tool  consisting  of  19  dimensions.  NCAPS  was  administered  to 
incoming  and  current  7251s. 

Validity  Analysis 

Validity  refers  to  the  correlation  between  a  predictor  of  success  (such  as  the  ASVAB 
GT  Score,  or  NCAPS)  and  an  objective  performance  measure  (such  as  school  grades  or 
graduation  status,  or  time  to  7257  qualification).  The  ASVAB  (GT  Score)  has 
demonstrated  validity  for  predicting  job  performance,  but  is  most  predictive  of 
academic  performance.  NCAPS  has  demonstrated  validity  for  predicting  job 
performance.  The  two  predictors  generally  do  not  correlate,  but  it  depends  upon  the 
constructs  being  measured  by  the  specific  criterion. 

Performance  Measures 

Job  performance  is  a  multifaceted  construct  that  can  be  measured  in  a  variety  of 
ways  (peer  ratings,  supervisor  ratings,  customer  ratings,  output,  qualifications,  absence 
of  errors,  etc.).  For  Phase  II,  a  variety  of  proxies  for  job  performance  were  considered, 
but  the  following  provided  the  most  significant  results. 
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Schoolhouse  performance  measures 


GPA  scores  for  Block  One,  Block  Two,  Block  Three,  and  Blocks  One  through  Three 
(1-3)  Cumulative  GPA  were  collected  and  standardized.  ASVAB  (GT  Scores)  were  also 
collected. 


Results 

The  resulting  data  from  the  ATC  schoolhouse  were  cleaned  and  subjected  to  various 
analyses  including  validity  analysis.  This  section  describes  the  data  obtained  and 
analyses  with  the  results  reported  up  front.  Validity  analysis  determined: 

•  NCAPS  scores  are  a  valid  predictor  of  success  for  Marine  Air  Traffic  Controllers 
in  the  schoolhouse.  Success  in  the  schoolhouse  is  determined  by  Block  1-3 
Cumulative  GPA. 

•  ASVAB  GT  Scores  correlated  significantly  with  GPA  for  the  schoolhouse  as  did 
NCAPS  measures. 

•  Regression  analyses  clearly  indicate  that  non-cognitive  measures,  especially  those 
assessed  with  the  Marine  Air  Traffic  Control  Suitability  Test  A  and  B  (MATC-ST 
A  and  B),  add  incremental  validity  above  the  ASVAB  GT  Score. 

Background  and  demographics 

This  section  describes  the  sample  of  participants  who  took  part  in  the  study,  as  well 
as  the  characteristics  of  the  performance  measures  collected  from  those  participants. 
Table  1  presents  background  and  demographic  information  for  209  Marine  Corps  ATC 
Schoolhouse  (7251)  participants.  The  majority  of  the  sample  was  male  (92.3%),  White 
(67%),  with  less  than  4  years  of  service  (91.87%).  Most  of  the  sample  was  Private  First 
Class  (PFC;  E2  paygrade),  (70.8%). 
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Table  1 

ATC  Schoolhouse  Demographics 


Frequency 

Percent 

Cumulative 

Percent 

Gender 

Male 

193 

92.3 

92.3 

Female 

16 

7.7 

100.0 

White 

140 

67.0 

67.0 

Hispanic 

30 

14.4 

81.4 

Ethnicity 

Black 

19 

9.1 

90.5 

Asian  or  Pacific  Islander 

8 

3.8 

94.3 

American  Indian 

2 

1.0 

95.3 

Other 

10 

4.8 

100.0 

El (PVT) 

21 

10.0 

10.0 

E2  (PFC) 

148 

70.8 

80.9 

Rank 

E3  (LCPL) 

8 

3.8 

84.7 

E4  (CPL) 

19 

9.1 

93.8 

E5  (SGT) 

13 

6.2 

100.0 

YoS 

>4 

192 

91.87 

91.87 

5-10 

17 

8.13 

100.0 

NCAPS  Data  Summary 

At  the  time  of  this  report,  complete  data  was  obtained  from  the  NCAPS  measure  for 
209  Marine  Corps  ATC  Schoolhouse  (7251 MOS)  participants. 

NCAPS  is  a  computerized  adaptive  personality  measure  consisting  of  19  components, 
four  of  which  were  related  to  attributes  relevant  to  the  current  research  effort.  For  each 
NCAPS  component,  the  data  consist  of  a  theta  value  that  describes  the  participants 
standing  on  the  construct  being  measured,  and  a  PSD  or  posterior  standard  deviation, 
which  is  an  index  of  algorithm  convergence.  A  theta  value  is  defined  as  an  individual’s 
standing  on  a  construct  being  measured  using  adaptive  testing,  or  item  response  theory, 
where  items  are  presented  to  individuals  based  on  their  responses  to  prior  items.  This 
process  continues  until  the  algorithm  closes  in  on  that  individual’s  standing  on  the 
construct,  which  is  represented  by  the  value  of  theta. 

There  were  seven  relevant  NCAPS  dimensions  for  the  ATC  schoolhouse 
performance.  These  NCAPS  dimensions  were  used  to  develop  the  algorithms  for  the 
Marine  Air  Traffic  Control  Suitability  Test  (MATC-ST).  There  are  two  relevant 
algorithms  for  the  ATC  schoolhouse  performance.  These  will  be  referred  to  as  the 
Marine  Air  Traffic  Control  Suitability  Test  Alpha  (MATC-ST  A)  and  the  Marine  Air 
Traffic  Control  Suitability  Test  Bravo  (MATC-ST  B).  The  Marine  Air  Traffic  Control 
Suitability  Test  Alpha  dimensions  were:  Achievement  Motivation,  Initiative, 
Perceptiveness/Depth  of  thought.  Social  Orientation,  and  Self-Reliance.  The  Marine  Air 
Traffic  Control  Suitability  Test  Bravo  dimensions  were:  Achievement  Motivation, 
Empathy,  Perceptiveness/Depth  of  Thought,  Self-Reliance,  and  Vigilance.  The  MATC- 
ST  B  has  been  included  in  this  report  because  it  incorporated  dimensions  that  overlap  in 
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content  to  the  significant  predictors  for  operational  performance  developed  in  Phase  I  of 
this  research  project.  The  degree  of  relevance,  or  relatedness  (validity)  was  derived 
through  regression  analyses,  which  identified  these  dimensions  (or  traits)  as  statistically 
significant  predictors  of  ATC  performance  at  the  Schoolhouse. 

ATC  Schoolhouse  Performance  Data  Summary 

The  ATC  Schoolhouse  performance  data  summary  in  Table  2  includes  the 
minimums,  maximums,  means,  modes,  and  standard  deviations  of  GT  Score,  7251  Block 
1 GPA,  7251  Block  2  GPA,  7251  Block  3  GPA,  and  7251  Cumulative  Block  1-3  GPA.  GT 
Scores  were  collected  for  208  ATC  schoolhouse  Marines,  with  a  minimum  GT  Score  of 
107,  a  maximum  GT  Score  of  139,  a  mean  GT  Score  of  118.23,  the  mode  GT  Score  was 
114,  and  the  standard  deviation  for  GT  Score  was  7.55. 

The  7251  Block  1  GPA  was  collected  for  140  ATC  schoolhouse  Marines,  with  a 
minimum  Block  1  GPA  of  73.80%,  a  maximum  Block  1  GPA  of  95.29%,  a  mean  Block  1 
GPA  of  84.93%,  the  mode  Block  1  GPA  was  78.35%  and  the  standard  deviation  for  Block 
1  GPA  was  4.99%.  The  7251  Block  2  GPA  was  collected  for  119  ATC  schoolhouse 
Marines,  with  a  minimum  Block  2  GPA  of  77.08%,  a  maximum  Block  2  GPA  of  95.04%, 
a  mean  Block  2  GPA  of  85.86%,  the  mode  Block  2  GPA  was  82.40%  and  the  standard 
deviation  for  Block  2  GPA  was  4.15%.  The  7251  Block  3  GPA  was  collected  for  100  ATC 
schoolhouse  Marines,  with  a  minimum  Block  3  GPA  of  77.98%,  a  maximum  Block  3 
GPA  of  95.95%,  a  mean  Block  3  GPA  of  87.95%,  the  mode  Block  3  GPA  was  90.00%  and 
the  standard  deviation  for  Block  3  GPA  was  3.70%.  The  7251  Block  1-3  Cumulative  GPA 
was  collected  for  100  ATC  schoolhouse  Marines,  with  a  minimum  Block  1-3  Cumulative 
GPA  of  77.28%,  a  maximum  Block  1-3  Cumulative  GPA  of  94.57%,  a  mean  Block  1-3 
Cumulative  GPA  of  86.26%,  the  mode  Block  1-3  Cumulative  GPA  was  83.99%  and  the 
standard  deviation  for  Block  1-3  Cumulative  GPA  was  3.94%. 


Table  2 

Operating  Forces  Performance  Descriptive  Statistics 


N 

Minimum 

Maximum 

Mean 

Mode 

Std. 

Deviation 

GT  Score 

208 

107 

139 

118.23 

114 

7.55 

Block  1  GPA 

140 

73.80% 

95.29% 

84.93% 

78.35% 

4.99% 

Block  2  GPA 

119 

77.08% 

95.04% 

85.86% 

82.40% 

4.15% 

Block  3  GPA 

100 

77.98% 

95.95% 

87.95% 

90.00% 

3.70% 

Block  1-3 
Cumulative 

100 

77.28% 

94.57% 

86.26% 

83.99% 

3.94% 

GPA 
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Marine  ATC  Suitability  Test  Score  and  Validity  Analysis 

The  Marine  ATC  Suitability  Test  (MATC-ST)  Alpha  and  Bravo  score  for  the  ATC 
schoolhouse  population  is  calculated  from  dimensions  of  NCAPS  that  were  submitted  in 
regression  analysis  with  the  ATC  schoolhouse  Block  1-3  Cumulative  GPA  as  the 
performance  criterion.  This  section  describes  the  correlation  and  regression  analyses. 

Table  3  presents  the  intercorrelations  of  the  predictor  test  scores,  (MATC-ST  A), 
(MATC-ST  B),  ASVAB  GT,  and  the  criterion,  ATC  schoolhouse  Block  1  GPA,  Block  2 
GPA,  Block  3  GPA,  and  Blocki-3  Cumulative  GPA. 


Table  3 

Predictor  Test  Scores  and  Performance  Intercorrelations 


MATC 

-ST  A 

MATC 

-ST  B 

GT 

Score 

Block 

1  GPA 

Block 

2  GPA 

Block 

3  GPA 

Cum 

GPA 

MATC-ST  A 

1 

MATC-ST  B 

.755** 

1 

GT  Score 

.129* 

.157* 

1 

Block  1  GPA 

.368** 

.293** 

.341** 

1 

Block  2  GPA 

.331** 

277** 

27]^** 

.858** 

1 

Block  3  GPA 

.399** 

.353** 

.328** 

.753** 

.860** 

1 

Block  1-3 
Cumulative  GPA 

.456** 

.373** 

.368** 

.939** 

.959** 

gj^2** 

1 

*  Correlation  is  significant  at  the  0.05  level  (2-tailed). 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 

Table  3  shows  the  intercorrelations  between  GT  Score,  the  MATC-ST  A  and  MATC- 
ST  B,  and  the  Schoolhouse  performance  metrics.  All  GPA  scores  were  significantly 
correlated  with  one  another,  with  both  Suitability  algorithms,  and  with  GT  Score  at  the 
p<.oi  level.  GT  Score  was  also  significantly  correlated  to  MATC-ST  A  and  MATC-ST  B  at 
the  p<.05  level.  For  completeness,  the  data  were  used  in  a  hierarchical  regression. 

Through  hierarchical  regression,  the  prediction  of  the  MATC-ST  A  and  B  score  on 
performance  outcomes  may  be  isolated  and  compared  to  the  prediction  of  other  factors, 
such  as  the  ASVAB  score,  as  well  as  the  prediction  of  composites  of  multiple  factors  at 
once.  This  technique  uses  comparisons  of  successive  regression  models  and  determines 
the  significance  that  each  one  has  above  and  beyond  the  others.  Three  models  were 
tested  and  compared:  (1)  the  ASVAB  GT  scores’  prediction  of  Cumulative  GPA  scores, 

(2)  the  MATC-  Suitability  Test  scores’  prediction  of  Cumulative  GPA  scores,  and  (3) 
regression  weighted  composite  scores’  (made  up  of  ASVAB  GT  and  MATC-ST)  of 
Cumulative  GPA  scores. 

The  Marines  currently  use  the  ASVAB  as  a  selection  screening  tool  to  gain  access  to 
the  ATC  School;  therefore,  GT  Score  was  the  first  variable  entered  (model  1),  followed  by 
the  MATC-ST  A  score  (model  2),  and  the  resulting  composite  (model  3).  The  same 
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model  process  was  conducted  for  MATC-ST  B  score  (model  2a)  and  the  resulting 
composite  (model  3a).  Results  of  the  regression  analysis  are  presented  in  Table  4 
(weights  are  standardized  beta  (P)  weights  and  thus  represent  correlations). 


Table  4 

ATC  Schoolhouse  Hierarchical  Regression  Validity  Analysis 


Measure 

Standardized 
Regression  Coefficient 

R2 

F 

t 

GT  Score 

.368 

.135 

15.237* 

3.914* 

MATC-ST  A 

.456 

.208 

25.696* 

5.066* 

GT  Score  & 

.296 

.293 

21.611* 

3.416* 

MATC-ST  A 

.403 

4.649* 

MATC-ST  B 

.373 

.139 

15.856* 

3.979* 

GT  Score  & 

.316 

.237 

12.895* 

3.519* 

MATC-ST  B 

.323 

3.591* 

a.  Dependent  Variable:  Block  1-3  Cumulative  GPA; 
p<.001  probability  level. 

^Statistically  significant  at  the 

The  Table  4  results  showed  that,  as  expected  from  the  results  in  table  3,  GT  Score 
alone  (Model  1)  predicts  Schoolhouse  performance  at  a  statistically  significant  level 
(Model  1;  P=.368,  R2=.i35,  F=15.237,  t  =3.914,  p<.ooo),  accounting  for  13.5%  of  the 
variance  in  performance.  When  the  MATC-ST  A  was  added  into  the  next  step  (Step  2)  of 
the  hierarchical  regression  analysis,  the  results  were  also  statistically  significant  (Model 
2;  P=.456,  R2=.2o8,  F=25.696,  t=5.o66,  p<.ooo).  Together,  the  MATC-ST  A  and  GT 
Score  account  for  29.2%  of  the  variance  in  performance. 

A  composite  of  the  two  variables  entered  in  the  third  model  (Step  3)  improved  the 
prediction  over  MATC-ST  A  alone  (Model  3;  p=.54i,  R2=.293,  F=2i.6ii,  1=4.646, 
p<.ooo).  Given  that  GT  Score  and  the  Suitability  Alpha  test  score  were  significantly 
correlated  (R2=.293),  this  finding  suggests  that  the  predictive  validity  of  the 
components  work  together  additively  to  increase  the  validity  of  the  prediction  with 
regard  to  Schoolhouse  performance. 

In  Table  4  results  showed  that,  the  MATC-ST  B  predicts  Schoolhouse  performance  at 
a  statistically  significant  level  (Model  2a;  P=.373,  R2=.i39,  F=i5.856, 1=3.979,  p<.ooo), 
accounting  for  14%  of  the  variance  in  performance.  When  the  GT  Score  and  MATC-ST  B 
were  hierarchically  regressed  onto  Schoolhouse  performance,  the  results  were  also 
statistically  significant  (Model  3a;  P=.487,  R2=.237,  F=i2.895, 1=3.588,  p<.ooo). 

Together,  the  MATC-ST  B  and  GT  Score  account  for  23.6%  of  the  variance  in 
Schoolhouse  performance.  Given  that  GT  Score  and  the  MATC-ST  B  score  were 
significantly  correlated  (R2=.237),  this  finding  suggests  that  the  predictive  validity  of  the 
components  work  together  additively  to  increase  the  validity  of  the  prediction  with 
regard  to  Schoolhouse  performance. 
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Figure  i  provides  a  graphical  representation  of  the  predictive  relationship  between 
GT  Score  and  Cumulative  GPA  score  (R2=.i35).  Figure  2  provides  a  graphical 
representation  of  the  predictive  relationship  between  MATC  Suitability  Test  A  score  and 
Cumulative  GPA  score  (R2=.2o8).  Figure  3  provides  a  graphical  representation  of  the 
predictive  relationship  between  MATC  Suitability  Test  B  score  and  Cumulative  GPA 
score  (R2=.i39). 

Figure  4  applies  to  the  combined  GT  and  MATC-Suitability  Test  A  as  the  predictor 
(R2=.293).  Figure  5  applies  to  the  combined  GT  and  MATC-Suitability  Test  B  as  the 
predictor  (R2=.237).  R^  (variance  overlap)  is  shown  on  each  graph  and  is  the  proportion 
of  the  Cumulative  GPA  score  that  is  accounted  for  by  the  predictor.  The  square  root  of 
this  value  is  the  correlation. 


Figure  1.  GT  Score  and  Cumulative  GPA 
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Figure  2.  MATC-ST  A  and  Cumulative  GPA 


11 


Cumulative  GPA 


Linear  =  0.139 


75.000- 


-1.50 


-1.00 


-.50  .00 

MATC-STB 


.50 


1.00 


Figure  3.  MATC-ST  B  and  Cumulative  GPA 
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Figure  4.  GT  Score  and  MATC-ST  A  Composite  and  Cumulative  GPA 
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R2  Linear  =  0.237 
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Figure  5.  GT  Score  and  MATC-ST  B  Composite  and  Cumulative  GPA 

Together,  GT  Score  and  MATC-  Suitability  Screening  Test  score  account  for  29.2%  of 
the  variance  in  schoolhouse  performance.  The  “variance  accounted  for”  metric 
interpretation  of  a  correlation  provides  meaningful  insights  into  the  value  of  a  selection 
&  classification  instrument.  That  is,  predictor  test  developers  strive  for  100% 
performance  variability  accounted  for  by  the  predictor  (a  correlation  of  1.0).  However, 
to  set  a  cutscore  for  an  operational  selection  standard,  the  correlation  coefficient 
(validity  coefficient)  is  required.  Setting  relevant  cutscores  will  be  further  analyzed  in 
Phase  III  when  the  data  from  the  next  cohort  going  from  the  ATC  schoolhouse  to  7257 
qualification  (Phase  III)  can  be  analyzed. 

The  MATC-ST  A  and  B  has  been  shown  to  be  predictive  of  success  for  Marine  ATCs 
in  the  schoolhouse.  Success  is  determined  by  Block  1-3  Cumulative  GPA,  therefore  the 
higher  the  MATC-ST  score  the  higher  the  Cumulative  GPA  of  the  ATC  schoolhouse 
graduate  (7251).  The  expectation  is  that  the  relationship  will  hold  up  for  future  ATC 
Marines  at  the  schoolhouse,  and  that  a  near  term  goal  would  be  to  evaluate  the  positive 
effects  of  an  operational  cutscore. 

An  additional  topic  of  interest  is  the  positive  impact  that  the  MATC-ST  had  on 
diversity.  From  the  ATC  schoolhouse  sample  (n=209),  we  sorted  GT  Scores  from  highest 
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to  lowest  and  we  selected  the  top  lOO  GT  scorers.  The  diversity  picture  is  illustrated  in 
Figure  6  and  annotated  here:  White  (75%)  and  Non-white  (25%);  Male  (95%)  and 
Female  (5%). 

We  then  sorted  the  GT  &  MATC-ST  A  scores  highest  to  lowest  and  we  selected  the 
top  100  scorers.  The  diversity  picture  is  illustrated  in  Figure  7  and  annotated  here: 
White  (70%)  and  Non-white  (30%);  Male  (96%)  and  Female  (4%).  We  also  sorted  the 
GT  &  MATC-ST  B  scores  highest  to  lowest  and  we  selected  the  top  100  scorers.  The 
diversity  picture  is  illustrated  in  Figure  8  and  annotated  here:  White  (60%)  and  Non¬ 
white  (40%);  Male  (91%)  and  Female  (9%). 


0  50  100 


Figure  6.  Top  100  GT  Score  and  Diversity 
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ElNon-White-30% 
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□  Male-96% 
■  Female-4% 


Figure  7.  Top  100  GT  Score  &  MATC-ST  A  and  Diversity 
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Figure  8.  Top  100  GT  Score  &  MATC-ST  B  and  Diversity 


The  addition  of  the  MATC-ST  can  improve  the  ATC  screening  process  through  fair, 
valid  screening  improvements  that  not  only  predict  high  performance  in  the  operating 
forces  (Walker,  Farmer,  &  Roberts,  2012)  and  schoolhouse,  but  also  increase  diversity 
within  the  Air  Traffic  Control  field. 


Recommendations 

At  the  time  of  this  report  the  Marine  ATC  Suitability  Screening  Test  (MATC-ST)  is 
shown  to  be  predictive  of  success  for  Marine  ATCs  in  the  operating  forces  (Phase  I 
results)  and  the  ATC  schoolhouse  (Phase  II  results).  The  Marine  ATC  Suitability  Test 
(MATC-ST)  score  is  a  valid  algorithm  derived  from  dimensions  of  NCAPS.  Further  data 
collection  and  analysis  following  the  ATC  schoolhouse  graduation  to  7257  qualification 
will  provide  confirmation  of  the  MATC-ST’s  predictive  validity,  both  for  training 
performance  and  ATC  job  performance.  It  is  best  practice  to  combine  concurrent 
validity  with  predictive  validity  from  the  ATC  Schoolhouse  in  order  to  recommend  any 
conservative  cutscores  and/or  implementation  into  the  Marine  Corps  Air  Traffic  Control 
selection  and  classification  process.  (This  recommendation  has  been  submitted  as  a 
Phase  III  proposal  and  is  pending  at  the  time  of  this  report). 

As  of  the  date  of  this  report  Phase  I  and  Phase  II  results  were  pending  to  be  briefed 
to  project  officers.  It  is  important  to  note  that  Phase  I  results  were  from  operating  forces 
and  Phase  II  results  were  from  the  schoolhouse.  Phase  III  will  look  at  the  best  overall 
MATC-ST  algorithm  for  predicting  Marine  ATC  performance  as  defined  by  project 
officers  (e.g.  time  to  qualify,  GPA,  behaviorally  anchored  rating  scales,  etc.). 

If  the  MATC-ST  is  to  be  utilized  for  selection/classification  it  is  recommended  that  it 
be  used  only  in  addition  to  the  ASVAB  (GT),  and  implemented  at  the  Military  Entrance 
Processing  Station  (MBPS)  prior  to  a  Marine  ATC  MOS  designation/assignment.  This 
will  reduce  attrition  and  increase  the  quality  of  Marines  selected  for  the  ATC  MOS. 
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Limitations 


The  length  of  this  study  was  approximately  one  year,  however  an  extension  was 
granted  for  more  data  collection  at  the  schoolhouse.  A  hindering  factor  was  that  data 
collection  at  the  schoolhouse  started  about  six  months  later  than  the  operating  forces. 
The  timing  difference  did  not  have  an  impact  on  results  other  than  delaying  Phase  II 
analysis  six  months  in  order  to  allow  enough  time  for  performance  measures  to  be 
collected.  Although,  NCAPS  allowed  for  maximum  participation  among  Marine  Air 
Traffic  Controllers  in  the  schoolhouse  because  it  was  web  enabled  and  administered  at 
the  beginning  of  training;  the  performance  data  collection  took  much  more  time  to 
collect.  It  was  necessary  for  an  adequate  amount  of  time  to  pass  to  collect  Block  One, 
Block  Two,  Block  Three,  and  Block  One  through  Three  (1-3)  Cumulative  GPA  data  from 
the  ATC  schoolhouse. 

Another  limitation  at  the  time  of  this  report  is  that  the  ATC  schoolhouse  data  and 
analysis  includes  cumulative  GPAs,  thus  failures  are  excluded.  Further  analysis  will  need 
to  be  performed,  as  recommended  for  Phase  III  to  allow  for  a  larger  sample  size  and  any 
range  restriction.  An  additional  report  would  need  to  be  produced  at  the  time  of  Phase 
III  MATC-ST  data  analysis  completion. 


Conclusion 

This  report  further  concluded  findings  from  the  ATC  operating  forces  study  (Phase  I) 
that  NCAPS  dimensions  calculated  specifically  for  the  Marine  Air  Traffic  Control 
community  can  predict  success  in  training  as  well  as  on  the  job.  Phase  II  of  the  Marine 
Air  Traffic  Control  Suitability  Screening  Test  (MATC-ST)  project  focused  on 
performance  at  the  ATC  schoolhouse,  predicting  higher  Cumulative  GPAs  in  ATC 
schoolhouse  graduates  (7251).  Recommendations  were  made  for  Phase  III  to  continue 
validation  of  MATC-ST.  The  expectation  is  that  the  relationship  will  hold  up  for  future 
ATC  Marines,  and  that  a  near  term  goal  would  be  to  evaluate  the  positive  effects  of  an 
operational  cutscore. 

MATC-ST  implementation  can  improve  the  Marine  ATC  screening  process  through 
better  schoolhouse  performance,  better  operating  forces  performance,  and  increased 
diversity  through  fair,  valid  screening  improvements.  As  with  any  test,  it  is  important  to 
continue  to  validate  these  tools  considering  changes  in  curriculum  and  additions  of 
simulation  based  training  in  order  to  see  if  the  ASVAB  composites  and  MATC-ST 
cutscores  are  adequate  for  producing  a  job  ready  Marine. 


17 


References 


Batram,  D.  (1993).  Emerging  trends  in  computer-assisted  assessment.  In  H.  Schuler, 
J.L.  Farr,  &  M.  Smith  (Eds.),  Personnel  seleetion  and  assessment:  Individual  and 
organizational perspeetives  (Chapter  17,  p.  267-288). 

Equal  Employment  Opportunity  Commission.  (1978).  Uniform  Guidelines  on  Employee 
Selection  Procedures.  Department  of  Labor  and  Department  of  Justice.  Washington, 

D.C. 

McBride,  T.M.  (2012).  Study  of  personnel  attrition  and  revocation  within  U.S.  Marine 
Corps  Air  Traffic  Control  specialties.  (Unpublished  master’s  thesis).  Embry-Riddle 
Aeronautical  University,  Daytona  Beach  EL. 

Principles  for  the  Validation  and  use  of  personnel  selection  procedures.  (1987,  3rd  ed). 
Society  for  Industrial  and  Organizational  Psychology.  College  Park,  MD. 

Walker,  K.M.,  Farmer,  W.L.,  and  Roberts,  R.C.  (2012).  Suitability  Screening  Test  for 
Marine  Corps  Air  Traffic  Controllers  Technical  Report  (Technical  Report  No.  13-1). 
Millington,  TN:  Navy  Personnel  Research  Studies  and  Technology. 


18 


Distribution  List 


AIR  UNIVERSITY  LIBRARY 

ARMY  RESEARCH  INSTITUTE  LIBRARY 

ARMY  WAR  COLLEGE  LIBRARY 

CENTER  EOR  NAVAL  ANALYSES  LIBRARY 

HUMAN  RESOURCES  DIRECTORATE  TECHNICAL  LIBRARY 

JOINT  EORCES  STAEE  COLLEGE  LIBRARY 

MARINE  CORPS  UNIVERSITY  LIBRARIES 

MARINE  CORPS  COMBAT  DEVELOPMENT  COMMAND,  TRAINING  AND 
EDUCATION  COMMAND  (TECOM) 

MARINE  CORPS  COMBAT  DEVELOPMENT  COMMAND,  OPERATIONS  ANALYSIS 
DIVISION  (OAD) 

NATIONAL  DEEENSE  UNIVERSITY  LIBRARY 
NAVAL  HEALTH  RESEARCH  CENTER 

NAVAL  POSTGRADUATE  SCHOOL  DUDLEY  KNOX  LIBRARY 

NAVAL  RESEARCH  LABORATORY  RUTH  HOOKER  RESEARCH  LIBRARY 

NAVAL  WAR  COLLEGE  LIBRARY 

NAVY  PERSONNEL  RESEARCH,  STUDIES,  AND  TECHNOLOGY  SPISHOCK 
LIBRARY 

OEEICE  OP  NAVAL  RESEARCH  (CODE  34) 

PENTAGON  LIBRARY 

USAE  ACADEMY  LIBRARY 

US  COAST  GUARD  ACADEMY  LIBRARY 

US  MERCHANT  MARINE  ACADEMY  BLAND  LIBRARY 

US  MILITARY  ACADEMY  AT  WEST  POINT  LIBRARY 

US  NAVAL  ACADEMY  NIMITZ  LIBRARY 


